HBASE-28836 Parallize the file archival to improve the split times #6616
+81
−31
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@virajjasani @stoty I am giving it another try. I hope this doesn't fail this time. I am running tests locally and they haven't finished yet. But i am optimistic this time. 🤞🏾 |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
I still see some failures that are not the usual flakies. |
mnpoonia
force-pushed
the
parallel_HBASE-28836
branch
from
c9022c1 to
4bd65e7
Compare
mnpoonia
force-pushed
the
parallel_HBASE-28836
branch
from
4bd65e7 to
d3b481d
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@stoty No test failure this time. Please have a look. Let me know if i am missing something here. |
stoty
reviewed
Jan 22, 2025
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
Show resolved
Hide resolved
| Queue<File> failures, String startTime) { | ||
| LOG.trace("Archiving {} files concurrently into directory: {}", files.size(), baseArchiveDir); | ||
|
|
||
| ExecutorService executorService = Executors.newCachedThreadPool(); |
There was a problem hiding this comment.
Most such similar pools are configurable.
Have you configured making the thread pool configurable ?
Would it make sense to use a global pool here, and limit the number of concurrent move operations ?
There was a problem hiding this comment.
Considering we are shutting down the threads after execution is it okay if we give some valid constant rather than a configuration? I am of the opinion that one more configuration would not help us. I also understand that having a max cap on number of threads is an important aspect.
There was a problem hiding this comment.
TBH I don't have a lot of experience with object storage deletion performance.
Are resolveAndArchive calls serial, or is it possible to have multiple invocations running at the same time ?
What do you think @wchevreuil, @BukrosSzabolcs ?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.